j 12.  IQ« 0  Vnv 


Report  No.  FAA-RD-80-35 


ff  F^  ■ 

y  F,  /  .  • 

*r  i  ■-  ^ 

*  ■  \  >  v*  •  v 


DETERMINATION  OF  THE  IMPACT  OF 
DIGITAL  DATA  BROADCAST  ON 
FLIGHT  TECHNICAL  ERROR 


Donald  W.  Richardson,  Ph.  D 


FEBRUARY  1980 
FINAL  REPORT 


DTIC 

,  Y-  *  .  •  '  m  I*  *'*■* 

,hi  t  '  ■ 


Document  is  available  to  the  U.S.  public  through 
the  National  Technical  Information  Service, 
Springfield,  Virginia  22161. 


Prepared  for 

U.S.  DEPARTMENT  OF  TRANSPORTATION 

FEDERAL  AVIATION  ADMINISTRATION 
Systems  Research  &  Development  Service 
Washington,  D.C.  2Q59C 


NOTICE 


This  document  is  disseminated  under  the  sponsorship 
of  the  Department  of  Transportation  in  the  interest 
of  information  exchange.  The  United  States  Government 
assumes  no  liability  for  its  contents  or  use  thereof. 


* 

I 


1 

l 


i 


S  s 


2.  Govt'nmvnt  Accession  No. 


iflo  ond  Subtitlo 


determination  of  the  ^impact  of  digital  ^ata 
^BROADCAST  ON ^FLIGHT  TECHNICAL^ERROR  ,  j 


l  •cnntcoi  Report  UocuiMfitwVto* 


3.  RKipitni  %  Cotolog  No. 


5.  Report  Dot* 

Februa*^  1980 


4.  Patterning  Organisation  Cad# 


8.  Perforating  Organisation  Report  No. 


/ Q yfconald  W ./Richardson)  Ph.D 


9.  Patterning  Organisation  Nome  and  Address 

Donrich  Research,  Incorporated 
1500  Oak  Drive 

West  Palm  Beach,  Florida  33406  ■  j 


12.  Sponsoring  Agency  Noma  and  Address 

Department  of  Transportation 
Federal  Aviation  Administration 
Systems  Research  and  Development  Service  ?- 
Washington,  D.C.  20590 


rniTETSmi 


10.  tttork  Unit  No.  (TRAIS) 


13.  Type  ot  Report  and  Period  Cowered 


Final  /depict 


Id.  Sponsoring  Agency  Code 

ARD-311 


16\Abstroct 


This  report  documents  the  data  reduction  and  analysis  of  existing 
flight  test  data  regarding  the  digital  data  broadcast  system  (DOBS) 
concept  of  automating  cockpit  data  input  procedures  in  an  area 
navigation  environment.  Particular  attention  is  paid  to  the  statistical 
quantification  of  the  impact  of  the  DDBS  concept  on  pilot  steering 
performance,  mainly  flight  technical  error  (FTE) . 

Results  of  this  analysis  indicate  that  DDBS  significantly  reduces 
both  pilot  blunder  rate  and  FTE,  for  both  the  enroute  and  approach 
phases  of  flight. 


17.  Ko y  Words 

Flight  Technical  Error, 
Digital,  Data,  RNAV, 
Blunders ,  Workload , 
Tacan,  DME 


19.  Security  Clottif.  (•(  Ait  report) 


HU  0S2 


18.  Distribution  Stotomont 

Document  is  available  to  the  U.S. 
public  through  the  National  Technical 
Information  Service,  Springfield,  Va. 

22161 


20.  Security  Clottif.  (of  Ihlt  page) 

UNCLASSIFIED 


UNCLASSIFIED 
Form  POT  F  1700.7  (8-72) 


Reproduction  of  completed  page  authorixod 


METRIC  CONVERSION  FACTORS 


Table  of  Contents 
List  of  Figures 
List  of  Tables 


TABLE  OF  CONTENTS 


Section 

1.0  INTRODUCTION 

1 . 1  GENERAL 

1.2  BACKGROUND 

1.3  OBJECTIVE 

2.0  EXPERIMENTAL  APPROACH 

2.1  INTRODUCTION 

2.2  DOBS  CONCEPT 

2.3  BASELINE  COMPARISON  CONCEPT 

2.4  EXPERIMENT  DESIGN 

2.5  DATA  REDUCTION  AND  ANALYSIS  TECHNIQUES 

3.0  FINDINGS  AND  RESULTS 

3.1  GENERAL 

3.2  BLUNDER  ANALYSIS 

3.3  FTE  ANALYSIS 

3.4  SUMMARY 

4.0  CONCLUSIONS  AND  RECOMMENDATIONS 

4.1  GENERAL 

4.2  CONCLUSIONS 

4.3  RECOMMENDATIONS  r 

Jli.0 •  •  1 

REFERENCES  V  - 

i  1  »  •* 

APPENDIX  A  j  i  -  - 

APPENDIX  B 


LIST  OF  FIGURES 


Figure  Page 


2.1  Controller-Modified  Route  Structure  2-4 

2.2  Routes  A-1  and  A-2  Horizontal  Profiles  2-5 

2.3  Northeast  RNAV  Departures  2-10 

2.4  Northeast  RNAV  Arrivals/Hotel  Transition  2-11 

2.5  Northeast  RNAV  Approach  Plate  2-12 

2.6  Southeast  RNAV  Departures  2-13 

2.7  Southeast  RNAV  Arrivals/Baltic  Transition  2-14 

2.8  Southeast  RNAV  Approach  Plate  2-15 

2.9  Northeast  RNAV  Departure  2-16 

2.10  Northwest  RNAV  Arrivals/Bay  Transition  2-17 

2.11  Northwest  RNAV  Approach  Plate  2-18 

2.12  NAFEC  ONE  RNAV  STAR  2-20 

2.13  Sample  "Quick  Look"  Airborne  Data  Printout  2-22 

2.14  Typical  EAIR  Tracking  Radar  Printout  2-23 

2.15  Ai rborne/ Radar  Data  Merge  File  2-25 

3.1  General  Aviation  RNAV  Control  Errors  by  Type 

(Baseline  Experiment)  3-7 


APPENDIX  A  -  EAIR  Plots 


A-1  thru  A-22 


11 


iHUBOlHaH 


'  -  *—• — <zzaiBsiema!ki 


LIST  OF  TABLES 


Table  Page 

2.1  Flight  Experience  of  Subject  Pilots  In  Hours  2-7 

3.1  DDB  Flight  Test  Matrix  Flown  by  Subject  Pilot  "A"  3-2 

3.2  DDB  Flight  Test  Matrix  Flown  by  Subject  Pilot  "B"  3-3 

3.3  DDB  Flight  Test  Matrix  Flown  by  Subject  Pilot  "C"  3-3 

3.4  DDB  Flight  Test  Matrix  Surrmary  of  Variables/Subject 

Pllot/Number  of  Flights  3-4 

3.5  DDB  Flight  Test  Pilot  Blunder  Analysis  3-6 

3.6  Procedural  Error  Rate  Per  Flight  3-7 

3.7  CDI  Aggregation  3-9 

3.8  DDBS-CDI  Aggregation  3-10 

3.9  FTE  Summary  (Baseline  vs  DDBS)  3-11 

3.10  FTE  Significance  Values  3-12 

3.11  FTE  Variance  Test  Variables  3-12 

APPENDIX  B  -  FTE  Data  B-l  thru  B-9 


111 


INTRODUCTION 


1.0 

1.1  GENERAL 

This  document  is  intended  to  serve  as  a  basic  data  report, 
primarily  as  an  aid  in  establishing  a  data  base  regarding  pilot 
performance  in  an  area  navigation  (RNAV)  operational  environment. 

The  two  principal  pilot  performance  measures  treated  in  this  report 
are  blunders  and  steering  or  flight  technical  error  (FTE).  The 
quantitative  data  contained  herein  is  directly  correlated  to,  and 
supplements,  the  more  qualitative  data  contained  in  Reference  1, 

Report  No.  FAA-RD-76-60,  "Evaluation  of  Digital  Data  Broadcast  for 
Area  Navigation".  In  order  to  effectively  evaluate  the  results 
and  conclusions  contained  in  this  present  document,  some  basic  material 
has  been  extracted  from  Reference  1  and  included  in  this  report  for 
clarity.  The  data  presented  in  this  report  has  been  processed  and 
analyzed  specifically  for  this  contract,  and  has  not  been  previously 
documented . 

1 .2  BACKGROUND 

At  the  present  time,  the  dramatic  expansion  of  avionics  technology 
is  giving  increased  emphasis  to  the  concept  of  area  coverage  navigation 
systems  as  a  companion  to,  if  not  a  replacement  for,  the  existing  V0R- 
based  airway  system.  Independent  of  the  specifics  of  the  navigation 
sensor  used  (VOR/DME,  Loran-C,  Omega/VLF,  GPS/NAVSTAR) ,  there  are 
serious  implications  regarding  cockpit  workload  and  pilot  steering 
performance  (sometimes  highly  correlated  with  a  measure  called  flight 
technical  error,  or  FTE)  that  must  be  considered  and,  if  possible, 
quantified  as  regards  eventual  certification  of  such  systems  for  use  in 
the  National  Airspace  System.  The  Federal  Aviation  Administration  (FAA) 

1$  continuously  involved  In  an  ongoing  program  of  analytical,  simulation 
and  flight  test  efforts  aimed  at  acquiring  a  substantive  quantitative  data 
base  relating  to  the  overall  subject  of  the  interrelationship  between 
pilot  performance  factors,  avionics  system  functional  design  criteria,  and 
overall  navigation  system  use  performance,  particularly  as  regards  avionics 
certification  standards,  ATC  operational  procedures  and  airspace  planning 
guidelines. 

Previous  FAA  activity  (Contract  No.  DOT- FA75WA- 3634)  had  sponsored 
a  hardware  development  program  and  flight  test  evaluation  of  a  concept 
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known  as  the  Digital  Data  Broadcast  System  (DDBS).  The  basis  of  this 
concept  involved  the  use  of  the  existing  Distance  Measuring  Equipment 
(DME)  channel  as  a  data  link  medium  for  transmitting  the  location  of 
RNAV  waypoints  comprising  complete  Standard  Instrument  Departure  (SID) 
and  Standard  Terminal  Arrival  Route  (STAR)  routes.  These  waypoints 
were  automatically  received  in  the  aircraft  and  used  by  a  low  cost  single 
waypoint  airborne  RNAV  computer  as  a  possible  means  of  minimizing  pilot 
workload  and  blunders  in  setting  the  series  of  waypoints  that  comprise 
the  SID  or  STAR  route  into  this  type  of  computer,  thus  potentially 
improving  the  pilot  steering  performance.  During  the  performance  of  the 
referenced  contract,  the  data  acquired  during  the  flight  tests  were 
reduced  as  regards  a  quantitative  comparison  of  blunder  performance  and 
a  subjective  evaluation  of  pilot  workload  reduction.  Data  relating  to 
the  quantitative  evaluation  of  FTE  was  originally  recorded  during  these 
tests  but  was  not  processed  or  analyzed  as  part  of  the  original  contract. 

1.3  OBJECTIVE 

It  is  the  purpose  of  this  effort  to  document  the  additional  data 
reduction  and  analysis  of  the  existing  DDBS  test  data  for  the  purpose 
of  statistically  quantifying  the  impact  of  the  DDBS  cockpit  data  input 
automation  concept  on  pilot  steering  performance,  namely  FTE.  The  results 
contained  herein  will  add  one  more  necessary  element  in  the  overall 
matrix  of  data  regarding  FTE  and  pilot  performance  for  area  coverage 
navigation  systems.  As  a  general  case,  the  blunder/workload  data  should 
be  able  to  be  extrapolated  to  other  area  navigation  sensor  systems 
offering  the  potential  for  cockpit  data  automation,  and  not  constrained 
to  the  specifics  of  VORTAC  navigation.  However,  the  absolute  impact 
of  DDBS  on  FTE  performance  must  be  carefully  considered  in  the  light  of 
VORTAC  signal  characteristics  and  their  possible  correlation  with  FTE 
as  observed  in  other  flight  test  experiments  (Reference  2). 

In  particular  it  must  be  noted  that  the  FTE  data  contained  in  this 
present  report  should  be  considered  primarily  in  the  context  of  a  direct 
comparison  with  previously  measured  FTE  data  taken  on  a  "baseline"  RNAV 
flight  test  experiment  as  reported  in  Reference  3.  As  presented  in  this 
report,  the  impact  of  DDBS  is  evaluated  in  terms  of  relative  improvement 
in  steering  performance  rather  than  as  an  absolute  value  of  FTE  per  se. 
Further  discussions  in  Section  3  of  this  report  will  amplify  this  point. 
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EXPERIMENTAL  APPROACH 


2.1  INTRODUCTION 

As  mentioned  previously  in  Section  1,  this  report  will  supplement 
and  amplify  the  qualitative  data  that  was  originally  presented  in 
Reference  1.  However,  in  order  to  properly  interpret  the  quantitative 
results  developed  in  this  present  report,  it  was  felt  necessary  to 
include  sufficent  information  to  describe  the  basic  test  environment, 
conditions,  procedures  and  equipment.  This  is  particularly  true  when 
the  comparison  is  ultimately  drawn  in  Section  3  between  the  DDBS  test 
results  and  the  FTE  values  obtained  from  the  non-DDBS  "baseline"  flight 
experiment.  For  this  reason  the  following  subsections  are  included. 

Some,  but  not  all,  of  the  experiment  description  has  been  extracted 
or  reconstructed  from  material  originally  presented  in  Reference  1. 

2.2  DDBS  CONCEPT 

T4l 

Preliminary  analysis  L  J  indicated  that  a  digital  data  broadcast 
system  (DOBS)  concept  could  be  used  and  applied  as  a  potential  means  of 
reducing  cockpit  workload  and  pilot  blunders.  The  initial  requirements 
of  a  DDBS  were  derived  from  an  operational  analysis  that  preceded  the 
design  of  DDB  engineering  model  hardware,  and  which  defined  an  operational 
concept  and  functional  requirements  for  that  system.  The  next  logical 
step,  following  the  fabrication  and  laboratory  testing  of  the  engineering 
model  DDBS,  was  the  evaluation,  under  actual  operational  flight  conditions, 
of  quantifiable  pilot  performance  parameters  as  affected  by  the  DDBS  units. 

As  described  in  Reference  4,  Digital  Data  Broadcast  is  a  technique 
in  which  the  DME  or  Tactical  Air  Navigation  (TACAN)  ground-to-air  radio 
link  is  used  to  carry  RNAV- referenced  data  into  the  aircraft.  In  the 
engineering  model  of  the  system  that  was  fabricated,  the  data  transmitted 
consisted  of  the  identification  code,  latitude,  longitude,  magnetic 
variation  and  elevation  of  the  transmitting  station,  and  the  RNAV  distances 
RNAV  bearings,  sequence  numbers  and  group  (route)  identifiers  of  the 
waypoints  referred  to  that  station.  These  latter  two  Items  of  information 
are  used  in  a  concept  of  transmitting  the  entire  sequence  of  RNAV  waypoints 
which  define  a  STAR,  such  that  the  pilot  may  select  only  a  STAR  designator 
in  order  to  acquire  the  entire  STAR  route.  The  digital  information, 
except  for  the  ground  station  identifiers,  then  is  decoded,  stored,  and 
displayed  to  the  pilot  by  the  DDBS  airborne  decoder.  The  applicable 
navigation  data  decoded  by  the  DDBS  airborne  equipment  then  are  fed  into 


the  RNAV  computer  as  required  for  computations  of  pertinent  RNAV  parameters. 
This  automatic  acquisition  of  RNAV  waypoint  data  accomplishes  two  major 
goals.  First,  no  pilot  workload  is  involved  in  setting  these  data  into 
the  computer,  since  it  is  done  automatically.  Second,  no  human  error  can 
be  introduced  by  missetting  coordinates  or  track  information. 

The  particular  DOBS  hardware  fabricated  for  this  program  was 
specifically  designed  to  be  interfaced  with  an  existing  single  waypoint 
analog  course  line  computer,  which  comprises  the  majority  of  those 
systems  currently  being  used  in  single-pilot  applications.  This  interface 
was  extremely  simple  and  necessitated  only  a  minor  modification  to  the 
existing  RNAV  computer  hardware. 

2.3  BASELINE  COMPARISON  CONCEPT 

The  history  of  the  flight  testing  of  area  navigation  devices  spans 
more  than  25  years.  For  a  variety  of  reasons  these  previous  flight  tests, 
while  providing  interesting  background  information,  were  not  particularly 
pertinent  to  the  problem  at  hand.  Specifically  those  tests  that  were 
run  prior  to  1972  did  not  necessarily  reflect  the  concepts  and  procedures 
currently  postulated  for  our  future  RNAV  environment.  Another  important 
issue  in  the  gathering  of  pertinent  data  on  a  particular  subject  involving 
experimentation  is  the  establishment  of  a  coordinated  concept  of  data 
collection,  in  which  the  types  of  data  and  the  experiments  themselves  are 
structured  on  a  common  basis,  such  that  data  from  one  experiment  to 
another  can  be  cross-correlated. 

It  has  been  a  matter  of  some  historical  interest  that  data  previously 
obtained  has  been  widely  scattered  in  the  type  and  quality  thereof, 
making  comparisons  between  systems  and  different  types  of  procedures 
extremely  difficult.  It  is  also  true  that  in  many  instances  in  the  past, 
the  types  of  data  that  have  been  collected  have  been  qualitative  or  sub¬ 
jective  in  nature.  Primarily,  It  is  interpretive  as  far  as  attitudes  by 
both  controllers  and  pilots  is  concerned.  A  more  quantitative,  systematic 
type  of  data  collection  and  data  analysis  needs  to  be  established  in  order 
to  get  meaningful  results  from  experiments  which  are  costly  and  often  few 
and  far  between. 


Using  lessons  of  the  past  as  a  guideline  and  the  Task  Force 
report  L  J  as  a  forcing  function,  a  concept  of  coordinated  data  collection 
was  initiated  and  has  been  successfully  applied  to  all  of  the  controlled 
RNAV  flight  tests  performed  since  1974.  The  particular  approach  that 
was  implemented  was  a  data  collection  end  flight  evaluation  program 
which  coordinates  cockpit  simulations,  experimental  flight  tests, 
operational  flight  tests,  and  real  time  terminal  area  simulations  to 
gather  and  evaluate  those  data  considered  necessary  in  order  to  reach 
an  intelligent  set  of  decisions  concerning  RNAV. 

The  critical  issue  that  has  been  kept  in  mind  during  the  planning 
phase  has  been  to  maintain  a  strict  discipline  concerning  commonality 
in  order  to  allow  extrapolation  and  cross-correlation.  Commonality  in 
this  case  does  not  always  mean  commonality  of  equipment,  since  different 
equipments  must  of  necessity  be  evaluated  in  order  to  establish 
differences  between  performance,  both  accuracy  and  functional.  The 
commonality  which  must  be  maintained  refers  to  the  more  ATC  environment 
related  issues,  such  as  commonality  of  routes  (both  enroute  and  terminal 
area),  commonality  of  procedures  (both  pilot  and  controller),  commonality 
of  the  uata  acquisition  concept  (parameters,  sampling  rate,  accuracy 
levels,  etc.)  and  finally,  the  commonality  as  far  as  the  techniques  of 
data  analysis  are  concerned,  so  that  the  same  assumptions,  ground  rules, 
compromises,  etc.,  are  applied  whereever  possible  to  each  of  the  data 
sources  that  are  used  to  establish  the  overall  data  base. 

In  order  to  realistically  evaluate  such  interrelated  and  involved 
elements  as  ATC  procedures,  controller  phraseology,  pilot/controller/ 
avionics  functions,  etc.,  it  was  decided  to  start  from  a  common  baseline 
of  routes  which  were  practical  from  an  airspace  design  point  of  view  - 
ones  that  were  in  concert  with  the  basic  guidelines  set  down  by  the 
Task  Force.  For  that  reason  one  basic  route  structure  was  selected  to 
be  the  common  element  in  several  of  the  later  simulation  and  flight 
experiments  performed  in  the  overall  effort  of  which  the  DDBS  experimental 
development  effort  was  one  important  element.  The  total  route  structure, 
which  is  shown  in  Figure  2.1,  was  basically  the  New  York  design  resulting 
from  the  analytical  effort  of  Reference  6,  with  principal  emphasis  on  the 
early  phases  of  RNAV  implementation  when  the  transition  from  a  VOR/radar 
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vector  environment  to  an  RNAV  environment  is  of  critical  importance. 

This  route  structure,  with  minor  variations  to  suit  the  individual 
circumstances,  was  used  in  the  real  time  simulations  of  References  7 
and  8,  the  cockpit  simulations  of  Reference  2,  and  the  baseline  flight 
test  of  Reference  3. 

In  the  flight  tests  of  Reference  3,  referred  to  as  the  "baseline" 
flight  tests,  the  same  basic  flight  paths  which  were  included  in  the 
referenced  simulations  were  used  to  form  a  series  of  RNAV  SIDs  and  STARs 
which  were  overlaid  over  the  NAFEC  airport.  A  light  twin  engine  aircraft 
(Aero  Commander  500)  using  a  simple  single  waypoint  general  aviation  type 
analog  RNAV  system  was  flown  under  simulated  instrument  flight  conditions, 
over  a  route  that  approximated  the  same  paths  that  would  be  flown  in  the 
New  York  terminal  area  in  an  RNAV  environment.  These  routes  are  shown 
in  Figure  2.2. 

For  the  DDBS  flight  experiments  which  formed  the  basis  of  this 
research  study,  a  continuing  and  concerted  attempt  was  made  to  maintain 
maximum  commonality  with  the  baseline  experiments  of  Reference  3.  In 
this  way  direct  comparisons  of  pilot  performance  could  be  made.  In  the 
DDBS  experiment,  the  following  items  were  either  similar  or  identical  to 
the  "baseline"  RNAV  flight  tests: 

Identical  between  each  test  (Aero  Commander  500) 
Identical  (NAFEC  terminal  area) 

Identical  between  each  test 
The  three  DDBS  subject  pilots  were  selected 
from  the  pool  of  the  original  six  subject 
pilots  of  the  "baseline"  tests 
DDBS  engineering  model,  modified  an  RNAV 
computer  identical  to  the  "baseline"  test  set 

The  DDBS  unit  was  located  in  the  same  place 
in  the  cockpit  as  was  the  RNAV  unit  in  the 
"baseline"  test 

Identical  between  each  test 
Identical  between  each  test 


Test  Aircraft  - 
Test  Location  - 
Test  Routes 
Test  Pilots 


RNAV  Equipment- 

Equipment 

Location 

Test  Instru¬ 
mentation 

Data  Processing 
and  Analysis  - 
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Fiqure  2.2  Routes  A-l  and  A-2  Horizontal  Profiles 
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In  this  manner,  a  direct  comparison  of  the  two  major  variables 
of  interest  to  this  study,  namely  the  numerical  values  of  blunders  per 
flight  and  the  statistical  measures  of  pilot  steering  performance  (FTE) 
can  be  made  between  the  "baseline"  tests  of  Reference  3  and  the 
subsequent  DDBS  experiment. 

2.4  EXPERIMENT  DESIGN 

While  the  basic  premise  of  this  original  DDBS  experiment  was  to 
evaluate  the  particular  features  of  the  DDBS  concept  as  embodied  by  the 
Engineering  Model  hardware,  every  attempt  was  made  to  maintain  a  maximum 
of  commonality  with  the  baseline  comparison  experiment  of  Reference  3. 
Three  subject  pilots  were  used  in  the  DDBS  flight  test  program  from  an 
original  pool  of  six  pilots  who  participated  in  the  baseline  experiment. 
All  of  the  subject  pilots  were,  therefore,  familiar  with  RNAV  operations 
and  the  flight  characteristics  of  the  test  aircraft,  and  were  responsible 
for  all  of  the  required  navigation  and  communication  tasks.  All  flights 
were  flown  simulating  IFR  conditions  by  using  an  inflight  training  hood. 
The  safety  pilot  was  the  designated  pilot  in  command,  and  would  inter¬ 
vene  only  for  traffic  avoidance  when,  and  if,  such  situations  arose. 

Each  subject  pilot  was  briefed  in  the  operational  use  of  the  tested  DDBS 
engineering  airborne  model  including  a  one  hour  orientation  flight  prior 
to  the  collection  of  flight  test  data.  Pilots  did  not  know  when  an 
impromptu  traffic  flow  clearance  would  be  given.  A  summary  of  the  subject 
pilots'  experience  level  is  presented  in  Table  2.1. 

Table  2.1  Flight  Experience  of  Subject  Pilots  in  Hours 


Subject 
Pi  1  ot 

Total 

Instrument 

Multi-Engine 

Light 

Twin 

RNAV 

A 

400 

70 

175 

175 

75 

B 

10,500 

700 

3,500 

300 

30 

C 

15,900 

700 

11,500 
-  -  .  ■■  ■  ^  . 

300 

45 

A  basic  DDBS  flight  test  program  was  designed  consisting  of  six 
dedicated  flight  tests  for  which  specific  objects  were  defined  as  follows: 
1.  Waypoint  Nomenclature  -  RNC  vs  OWN  --  Evaluate  the  relative 
merits  of  two  distinct  waypoint  identification  techniques. 

From  the  standpoint  of  operational  utility,  should  the 
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preferred  waypoint  designation  technique  be  the  route 
numbering  concept  (RNC)  in  which  the  pilot  enters  a 
single  five  character  data  entry  into  the  airborne  decoder 
which  designates  the  entire  selected  RNAV  arrival  or 
departure  route,  or  the  discrete  waypoint  numbering  concept 
( OWN)  in  which  the  pilot  enters  the  individual  code  number 
of  each  waypoint  he  wishes  to  use? 

2.  Traffic  Flow  Transition  --  Evaluate  and  isolate  problem  areas 
(operational  and  procedural)  in  the  terminal  area  by  using 
the  ODBS  concept  during  traffic  flow  (runway  in  use) 
changes  and  impromptu  sequences  within  a  broadcast  flow. 

3.  Waypoint  Sequences  -  Auto  vs  Manual  —  Determine  from  a 
design  requirement  viewpoint  whether  automatic  waypoint 
sequencing  is  a  desirable  system  feature. 

4.  Broadcast  Flow  Cycle  Time  --  Determine  the  operational 
acceptable  value  of  broadcast  cycle  time  from  a  minimum 
of  10  seconds  to  a  maximum  of  30  seconds. 

5.  Waypoint  Storage  —  Determine  the  optimum  or  acceptable 
number  of  waypoint  storage  registers. 

6.  VNAV  Impact  --  Determine  the  impact  of  VNAV  procedures  on 
the  DDBS  user,  concerning  such  variables  as  workload, 

blunder  reduction,  and  changes  in  aircraft  steering  performance. 

Overlaid  above  all  of  the  preceding  detailed  objectives,  which  were 
selected  to  optimize  the  design  features  of  an  ultimate  DDBS  system,  was 
the  basic  criteria  of  overall  blunder  performance,  pilot  steering 
performance  and  outer  loop  airspace  utilization. 

The  primary  DDBS  flight  test  program  consisted  of  flying  terminal 
area  SIDs  and  STARs.  A  total  of  three  test  patterns  of  waypoints  were 
transmitted  over  the  experimental  VORTAC  (VOR  colocated  with  TACAN). 

Each  of  the  test  patterns  contained  several  RNAV  departures  and  arrivals, 
but  all  were  referenced  to  a  specific  runway  in  use.  The  three  test 
patterns  were  the  following: 

1.  Northeast  (runway  04)  Departure/Arrival  (Figures  2.3 

thru  2.5) 

2.  Southeast  (runway  13)  Departure/ Arrival  (Figures  2.6 

thru  2.8) 

3.  Northwest  (runway  31)  Departure/Arrival  (Figures  2.9 

thru  2.11 ) 
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Four  different  routes  were  flown  on  the  three  combined  test  patterns. 

As  in  a  normal  operational  situation,  all  arrival  and  departure 
route  patterns  for  one  specific  traffic  flow  (i.e..  Northeast)  were 
broadcast  at  any  given  time.  This  corresponds  to  the  fact  that  while 
one  runway  Is  active  (in  use  for  arrivals  and  departures),  the  actual 
arrivals  and  departures  to  and  from  any  given  airport  can  operate  in 
any  direction.  In  the  situation  when  the  active  runway  is  changed 
(i.e.,  from  Northeast  flow,  runway  04  to  Northwest  flow,  runway  31), 
the  entire  broadcast  pattern  would  be  changed  (from  Figures  2.3  thru 
2.5  to  Figures  2.9  thru  2.11).  The  shaded  route  segments  on  each  test 
pattern  denote  the  particular  route  flown  on  that  specific  test  pattern. 

1.  Northeast  Pattern  --  (Figures  2.3  through  2.5)  --  The 
aircraft  would  be  flown  utilizing  the  Tango  Departure 
(024NE)  to  a  "direct- to"  transition  at  the  Tango  waypoint 
for  the  Victor  arrival  (A27NE)  to  a  landing  on  runway  04. 
Note  the  RNC  type  route  designator,  where  D  indicates 
departure  route,  24  indicates  the  center  of  the  departure 
octant  through  which  the  departure  will  be  made,  and  NE 
indicates  the  Northeast  runway  (04)  is  the  current  runway 
flow  in  use. 

2.  Southeast  Pattern  —  (Figures  2.6  through  2.8)  —  The 
aircraft  would  be  flown  utilizing  the  Tango  Departure 
(D24SE)  to  a  "direct-to"  transition  at  the  Tango  waypoint 
for  the  Victor  arrival  (A27SE)  to  a  landing  on  runway  13. 

2.  Northeast  Pattern  --  (Figures  2.9  through  2.11)  --  This 
pattern  was  utilized  whenever  an  impromptu  traffic  flow 
change  was  required.  The  aircraft  was  flown  as  in  the 
Northeast  pattern  (1)  until  halfway  between  Victor  and 
Gulf  waypoints.  At  that  point,  the  aircraft  was  cleared 
(impromptu)  for  the  Victor  arrival  (A27NW),  coinciding 
with  a  traffic  flow  change  by  the  broadcasting  VORTAC 
station  from  the  Northeast  to  the  Northwest  test  pattern. 

At  2.0  nautical  miles  "To"  Gulf,  one  of  two  new  clearances 
were  given;  proceed  direct  to  Somer  or  Port  waypoints.  If 
the  aircraft  went  by  way  of  the  Somer  waypoint,  a  new 
clearance  was  given  just  2.0  nm  short  of  it  for  a  direct-to- 
Day  waypoint.  No  new  clearances  (impromptu)  were  given  if 
the  aircraft  went  by  way  of  Port  waypoint.  Landings  on 
this  pattern  were  made  to  runway  31. 
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Figure  2.5  Northeast  RNAV  Approach  Plate 
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2.5  DATA  REDUCTION  AND  ANALYSIS  TECHNIQUES 

As  discussed  previously,  the  two  major  issues  concerning  this 
experimental  program  were  the  comparison  of  pilot  blunders  and  pilot 
steering  performance  between  the  baseline  RNAV  experiments  (non-DDBS) 
and  the  DDBS  experiments.  Each  of  these  issues  utilized  widely  differing 
data  acquisition  and  analysis  techniques  in  order  to  reach  particular 
conclusions,  and  therefore  they  are  each  briefly  explained  in  the  following 
paragraphs. 

During  the  flight  tests,  a  trained  cockpit  observer  monitored  and 
maintained  an  accurate  log  of  routine  and  special  events  that  occurred 
during  a  given  flight.  The  flight  logs  recorded  by  the  cockpit  observer 
proved  to  be  the  major  source  of  data  acquisition  from  which  blunder 
results  could  be  evaluated.  In  addition,  the  correlation  between  the 
observer  logs  and  EAIR  radar  plots  was  also  used  to  identify  the 
relative  seriousness  of  any  blunders  as  regards  airspace  utilization. 

Figure  2.12  indicates  typical  EAIR  radar  plot  results  from  the  baseline 
flight  tests  of  Reference  3,  with  indications  of  the  protected  airspace 
limits  and  annotations  from  the  observer's  logs.  The  major  issue  at 
stake  as  regards  this  current  effort  is  the  development,  either  through 
flight  hardware  or  cockpit  procedures,  of  a  means  of  improving  pilot 
blunder  performance  in  a  dynamic  flight  environment.  Automation  aids, 
such  as  the  DDBS,  may  very  well  alter  the  character  of  the  pilot's  role 
in  the  navigation  and  control  mission  he  has  traditionally  undertaken. 

Since  the  very  nature  of  the  pilot's  task  is  interpretive  and  adaptive, 
measurement  of  his  performance  as  regards  decision  making  and  attentiveness 
must  also  be  somehwat  qualitative  or  subjective. 

As  regards  the  statistical  quantification  of  pilot  steering 
performance  or  flight  technical  error,  a  process  identical  to  that 
carried  out  for  the  baseline  tests  was  utilized.  An  airborne  digital 
recorder  (Incre-Data  Mark  II)  recorded  the  following  parameters  on 
magnetic  tape.  Each  of  these  parameters  was  recorded  once  per  second. 

•  Time  (coordinated  with  EAIR  radar  time  for  correlation 

purposes) 

•  Crosstrack  Deviation 

•  Distance  to  Waypoint 

•  Altitude 
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NAFEC  ONE  RNAV  STAR 
North  Arrival  Echo  Waypoint 
95°  Left  Hand  Turn 
CD!  Sensitivity  =  1/4  nn\/dot 


Figure  2.12  NAFEC  ONE  RNAV  STAR 


I  Heading 

•  Vertical  Deviation 

•  DME  Distance  to  Station 

•  VOR  Bearing  to  Station 

•  Selected  Track  Angle 
'  •  Event  Marker 

Subsequent  to  each  flight  the  magnetic  tape  was  removed  from  the 
aircraft  and  run  through  a  software  program  which  converted  the  recordings 
to  engineering  units  (degrees,  feet,  miles,  etc.)  and  provided  a  "quick 
look"  printout  of  the  results  of  that  flight.  By  visual  inspection  of 
this  printout,  a  sample  of  which  is  shown  in  Figure  2.13,  an  initial 
determination  of  the  validity  and/or  reasonableness  of  the  data  can  be 
made.  In  this  way,  if  for  some  reason  the  data  was  not  valid  or 
acceptable,  the  test  could  be  rescheduled  or  the  deficiency  corrected 
before  the  test  program  was  continued. 

During  the  flight  the  EAIR  precision  tracking  radar  recorded  the 
actual  position  of  the  aircraft  on  magnetic  tape.  The  following  parameters 
were  recorded  ten  times  per  second  on  tape  and  printed  out  as  hard  copy 
immediately  subsequent  to  each  flight.  Figure  2.14  illustrates  a  typical 
EAIR  tracking  printout. 

9  Azimuth  (accurate  to  0.011°) 

•  Elevation  (accurate  to  0.011°) 

•  Range  (accurate  to  20  yards) 

•  Lati tude 

•  Longitude 

•  Hei ght 

•  Real  Time 

In  addition  to  the  recorded  position  of  the  aircraft,  a  visual 
trace  was  made  in  real  time.  The  traces  for  all  of  the  valid  data 
flights  run  during  this  test  program  are  contained  in  Appendix  A. 

These  plots  also  indicate  the  intended  aircraft  track  per  the  flight 
test  plan  for  that  specific  flight.  This  trace  allows  a  visual 
inspection  of  the  flight  for  correlation  with  the  observer's  logs.  Using 
these  plots,  certain  portions  of  the  EAIR  tracking  data  and/or  the  airborne 
Incre-Data  data  may  be  edited  out  as  being  invalid  or  not  pertinent  to 
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.13  Sample  "Quick  Look"  Airborne  Data  Printout 


Figure  2.14  Typical  EAIR  Tracking  Radar  Printout 


the  particular  situation.  In  particular,  since  the  RNAV  system  does 
not  provide  command  guidance  around  turns,  only  steady  state  (non¬ 
turning)  data  was  used  in  the  quantification  of  FTE.  After  data  editing, 
time  correlating  with  the  observer's  data  logs  and  establishing  start/ 
stop  of  valid  data,  the  EAIR  data  tape  was  time  merged  with  the  Incredata 
tape,  producing  the  Airborne/Radar  Data  Merge  File  shown  in  Figure  2.15. 

For  the  purposes  of  this  particular  analysis,  the  mean  and  standard 
deviation  of  the  crosstrack  deviation  (equivalent  to  FTE)  was  calculated 
from  every  ten  second  interval  of  data  considered  to  be  valid  from  the 
Data  Merge  File,  as  shown  in  Figure  2.15.  These  statistical  measures 
were  calculated  for  the  following  combi  nations  of  test  paramters: 

•  For  each  individual  route  segment  (track  between  any 
two  waypoints) 

•  For  each  flight  (aggregation  of  all  segments  comprising 
one  complete  flight) 

•  For  each  pilot  (aggregation  of  all  flights  flown  by 
each  subject  pilot) 

•  For  entire  test  (aggregation  of  all  flights  flown  by 
all  pilots) 

These  data  were  then  compared  to  the  results  for  the  baseline 
tests,  reported  in  Reference  3  in  an  identical  manner,  for  the  final 
resolution  of  the  quantitative  impact  of  the  DDBS  concept  of  cockpit 
data  input  automation  on  pilot  steering  performance.  Chapter  3  discusses 
the  results  of  the  analysis  of  both  the  blunder  data  and  the  FTE  data 
resulting  from  this  experiment. 


3.0 

3.1  GENERAL 


FINDINGS  AND  RESULTS 


As  discussed  previously  in  Reference  1,  the  flight  research  phase 
of  the  DOBS  test  program  consisted  of  thirty  six  (36)  flights  flown  at 
the  FAA  NAFEC  facility  at  Atlantic  City,  New  Jersey  during  the  test 
period  of  September  15,  1975  to  November  13,  1975.  Of  these  flights, 
twenty  six  US)  were  for  data  collection  purposes,  with  the  rest  devoted 
primarily  to  equipment  shakedown  or  pilot  orientation  purposes. 

Tables  3.1  through  3.3  show  the  actual  DDBS  flight  test  runs 
flown  by  each  of  the  three  subject  pilots,  whose  experience  level  has 
been  shown  in  Table  2.1.  Table  3.1  illustrates  that  portion  of  the  test 
matrix  flown  by  subject  pilot  A.  Notice  that  the  matrix  is  balanced 
among  the  sixteen  (16)  flights  as  regards  the  primary  test  variables 
discussed  in  Section  2.4  with  the  exception  of  the  waypoint  storage 
issue.  The  original  test  matrix  established  in  Reference  4  had  defined 
a  40%-60%  split  between  the  2-  and  6-  waypoint  test.  When  the  original 
two  pilot  matrix  had  to  be  perturbed,  for  scheduling  reasons,  to 
accomodate  an  additional  subject  pilot  (subject  pilot  C),  this  planned 
distribution  was  not  directly  achievable.  Tables  3.2  and  3.3  present 
that  portion  of  the  test  matrix  which  was  flown  by  subject  pilot  B  and 
C.  Table  3.4  shows  an  overall  summary  of  test  variables  per  subject 
pilot  per  number  of  flights.  Reference  to  Table  3.4  will  indicate  the 
relative  balance  of  the  final  DDBS  flight  test  matrix  as  actually  flown. 

In  each  of  the  previously  referenced  Tables  3.1  through  3.3,  the 
arrival /departure  designator  code  in  the  tables  (i.e.,  A27SE)  refers  to 
the  DDBS  nomenclature  code  of  an  arrival  or  departure  route  previously 
defined  as  part  of  a  broadcast  traffic  flow  pattern,  in  this  case  the 
Southeast  traffic  flow  for  runway  13  arriving  through  the  270°  arrival 
octant.  Likewise  a  D24NE  refers  to  a  departure  route  from  runway  04 
departing  through  the  240°  departure  octant  in  a  selected  and  broadcast 
traffic  flow.  The  remaining  major  headings  of  the  tables  merely  refer 
to  the  configuration  of  specific  test  variables  in  a  given  flight  test. 

3.2  BLUNDER  ANALYSIS 

As  discussed  in  Section  2,  a  major  factor  in  the  design  of  this 
DDBS  experiment  in  particular,  and  the  continuing  series  of  area 
navigation  pilot  performance  studies  in  general,  was  the  establishment 
of  a  maximum  of  commonality  among  experimental  variables.  In  order  to 
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assess  the  DOBS  potential  for  effecting  a  redaction  in  pi lot- induced 
blunders,  this  data  analysis  included  a  direct  comparison  of  blunder 
error  data  between  the  DOBS  flight  test  program  and  the  "baseline" 
general  aviation  RNAV  flight  test  previously  reported  in  Reference  3. 

This  comparison  is  particularly  valid  since  the  DDBS  test  aircraft, 
cockpit  configuration  and  basic  RNAV  system,  and  test  routes  were 
identical  or  similar  to  a  major  degree  to  the  baseline  exper  neot.  In 
addition  the  total  number  of  test  flights  were  nearly  equal  (30  for 
baseline  vs  26  for  DDBS),  and  most  importantly  the  three  DDBS  subject 
pilots  were  part  of  the  original  baseline  subject  pilot  population. 
Particularly  in  the  case  of  blunders,  where  data  was  recorded  on  an 
individual  pilot  basis,  a  direct  comparison  can  be  made  concerning  the 
impact  of  DDBS  on  blunder  performance  for  each  specific  pilot. 

Figure  3.1,  appearing  in  Reference  1,  shows  the  pilot's  RNAV  control 
errors,  by  type,  which  were  observed  dur.ng  the  previous  baseline 
tests.  An  examination  of  this  figure  indicated  that  thirty-one  (31) 
subject  pilot  errors  were  documented.  This  computes  to  an  average  value 
of  1.03  errors  per  flight.  In  the  twenty-six  (25)  DDBS  data  flights 
flown  by  the  three  subject  pilots,  only  four  pilot-induced  blunders  were 
recorded.  This  results  in  an  average  value  of  0.15  errors  per  flight, 
or  a  reduction  of  86%  in  pilot  blunders  which  could  be  attributed  to  the 
utilization  of  DDBS  in  the  RNAV  environment  of  the  terminal  area. 

A  summary  of  the  pilot  blunder  data  recorded  during  these  tests 
is  presented  in  Table  3.5.  An  analysis  of  this  data  indicates  that 
only  two  of  the  four  blunders  actually  resulted  in  a  violation  of 
protected  airspace.  Analysis  of  these  data  also  shows  that  pilot  A 
had  a  blunder  rate  only  0.06  errors  per  flight  as  opposed  to  0.17 
and  0.50  for  pilots  B  and  C,  respectively.  This  seeming  disparity 
could  very  well  be  attributed  to  the  fact  that  pilot  A  had  been  involved 
in  the  preliminary  analysis  and  design  of  the  DDBS  hardware  and  therefore 
was  much  more  familiar  with  its  theory  of  operation  than  the  other  two 
subject  pilots.  That  fact  notwithstanding,  even  the  poorest  pilot 
from  a  blunder  standpoint  had  an  error  rate  of  only  0.50  errors  per 
flight  as  compared  to  the  average  value  of  1.03  from  the  baseline  test. 


Pilot  Blunder  Analysis 


Error  Type 


1.  VORTAC  Frequency  4.  Waypoint  Distance  7.  Steering  Horizontal 

2.  CDI  Sensitivity  Change  5.  OBS  Course  8.  Stiering  Vertical 

3.  Waypoint  Radial  6.  Waypoint  Altitude  Set  9.  Wrong  Waypoint 

(VNAV) 

Figure  3.1  General  Aviation  RNAV  Control  Errors  by  Type  (Baseline  Experiment) 


Another  interesting  evaluation  of  blunder  data  is  to  compare  the 
value  of  blunders  per  flight  between  the  identical  pilots  from  the 
baseline  tests  of  Reference  3  and  the  current  DDBS  tests.  Table  3.6 
illustrates  the  fact  that  for  DOBS  subject  pilots  f  and  B,  their  rate 
of  blunders  per  flight  was  reduced  markedly  to  values  approximately 
1535  and  10%  respectively  of  the  baseline  values.  Subject  pilot  C  had  an 
increase  in  blunder  rate  per  flight  to  125%  of  his  original  baseline 
error  rate.  In  the  overall  sense,  pilot  C's  increase  does  not  appear 
to  have  the  statistical  significance  of  pilots  A  and  B's  drastic  error 
rate  reduction.  The  average  error  rate  for  the  identical  three  pilots 
decreased  to  17%  of  their  baseline  value,  again  a  significant  reduction. 


Table  3.6  Procedural  Error  Rate  Per  Flight 


DDBS 

Subject  Pilot 

Procedural  Error 

Rate  Per  Flight 

Basel ine 

DDBS 

A 

0.4 

0.06 

B 

1.8 

0.17 

C 

0.4 

0.50 

Average 


0.87 


0.15 


3.3  FTE  ANALYSIS 

The  second  quantifiable  measure  of  the  impact  of  the  DDBS  concept 
on  pilot  performance  is  the  comparative  evaluation  of  flight  technical 
error.  As  discussed  in  Section  2,  since  this  DDBS  flight  test 
experiment  was  designed  to  be  identical  to  the  experiment  described 
in  Reference  3  as  regards  flight  routes,  equipment,  instrumentation 
and  data  reduction,  a  direct  comparison  of  the  results  of  the  two 
experiments  can  be  made.  In  the  blunder  analysis  described  earlier  in 
this  chapter,  a  pilot-by-pilot  comparison  of  blunders  per  flight  was 
made.  In  the  case  of  FTE,  however.  Reference  3  did  not  document  FTE 
on  a  per-pilot  basis.  For  that  reason,  FTE  statistics  contained  herein 
will  be  compared  on  an  overall  test  data  basis.  It  will  be  shown  that 
the  data  sample  used  for  this  comparison  is  of  a  magnitude  that  the 
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comparisons  made  are  statistically  definitely  significant. 

As  outlined  in  Section  2,  the  deflection  of  the  course  deviation 
indicator  (CDI)  was  used  as  the  measure  of  pilot  steering  performance. 

The  pilot's  primary  control  function  was  to  maneuver  the  aircraft  to 
keep  the  CDI  needle  centered  at  all  times.  The  deflection  (if  any) 
of  that  needle  was  a  direct  measure  of  how  far  from  the  centerline 
of  the  desired  ground  track  (the  line  proceeding  into  the  "To"  or 
active  waypoint  at  the  selected  OBS  bearing)  the  RNAV  system  thinks 
it  is.  The  output  of  crosstrack  deviation  from  the  RNAV  computer 
drives  the  CDI  in  a  linear  manner.  That  is  to  say,  the  deflection  of 
the  CDI  needle  is  directly  proportional  to  the  linear  distance  off  track. 
For  the  DDBS  tests,  a  deflection  of  one  "dot"  (approximately  .125  inches) 
on  the  face  of  the  CDI  instrument  normally  indicated  a  crosstrack 
deviation  of  1.0  nm.  During  the  final  approach  portion  of  the  flight, 
normal  pilot  operating  procedure  was  to  switch  the  scale  sensitivity 
of  the  output  of  the  RNAV  computer  to  0.25  nm  per  dot. 

The  airborne  instrumentation  recorded,  as  one  of  its  parameters, 
the  electrical  signal  of  crosstrack  deviation  sent  from  the  RNAV 
computer  to  the  CDI  instrument.  This  value,  seen  as  the  CDI  column 
on  Figures  2.13  and  2.15,  was  used  to  compute  the  FTE  statistics  presented 
in  this  section.  As  discussed  previously,  the  CDI  value  at  every  10 
seconds  of  merged  data  was  originally  aggregated  into  data  sets  for 
each  DDBS  flight  segment  (between  two  waypoints),  and  then  further 
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combined  into  groups  of  flight  segments  categorized  as  "terminal" 
and  "approach"  phases  of  flight  for  comparison  with  the  baseline  data 
of  Reference  3.  Appendix  B  contains  the  original  data  sets  used  for 
these  aggregations.  Note  in  Appendix  B  that  the  number  of  data  samples 
(column  N)  is  shown  for  each  segment.  As  stated  in  Section  2,  some  data 
from  each  flight  was  selectively  edited  out  prior  to  statistical 
evaluation  for  a  variety  of  reasons.  Thus  the  data  samples  are  not 
identical  for  each  flight  over  the  same  course.  This  same  data  editing 
technique  was  applied  to  the  baseline  data  of  Reference  3  however,  so 
any.  comparisons  between  the  two  data  sets  can  be  made  on  the  same  basis. 
For  the  same  basic  reason,  two  flights  flown  by  subject  pilot  B  were  not 
included  in  the  FTE  data  set.  The  performance  of  the  instrumentation 
was  such  on  these  two  flights  that  the  data  was  not  considered  to  be 
valid.  Table  3.7  contains  the  results  of  the  initial  aggregation  of 
the  data  of  Appendix  B  combined  into  FTE  values  of  mean  (x)  and  two  sigma 
presented  on  a  per  flight  per  pilot  basis,  indicating  also  the  number  (N) 
of  10  second  data  samples  for  each  flight. 


Table  3.7  CDI  Aggregation 


Tota 

1  Per  Flight  Per  Pilot 

Flight  No. 

Pilot 

N 

X 

2a 

1 

A 

209 

.135 

1.442 

2 

A 

200 

.097 

1.572 

3 

A 

192 

-.015 

.758 

4 

A 

196 

-.066 

.708 

5 

A 

238 

-.141 

1.275 

6 

A 

262 

-.138 

1.068 

7 

A 

277 

.150 

.815 

8 

A 

256 

.203 

.712 

9 

A 

246 

.210 

.800 

10 

A 

192 

.259 

.897 

11 

A 

137 

.150 

.746 

12 

A 

127 

.075 

.694 

13 

A 

161 

.033 

.487 

14 

A 

188 

-.033 

.876 

15 

A 

216 

.052 

.711 

16 

A 

211 

-.094 

.792 

1 

B 

146 

-.120 

.635 

4 

B 

198 

-.201 

.612 

5 

B 

184 

.028 

.578 

6 

B 

204 

-.007 

.617 

1 

C 

205 

-.023 

.942 

2 

C 

257 

.045 

.955 

3 

C 

199 

-.086 

.825 

4 

C 

232 

.022 

.741 
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Further  statistical  aggregation  of  these  data  results  in  Table 
3.8,  when  the  FTE  performance  of  all  flights  flown  by  each  pilot  is 
presented  on  a  per  pilot  basis  and  then  as  a  total  FTE  statistic  for 
the  entire  DDBS  flight  test  program.  Two  incidental  comments  on  Table 
3.8  are  in  order.  It  is  interesting  to  note  that  While  the  blunder 
analysis  presented  previously  indicated  that  subject  pilot  A  had  the 
lowest  blunder  rate  per  flight,  this  same  subject  pilot  had  the  poorest 
FTE  performance.  This  could  have  resulted  from  the  fact  that  subject 
pilot  A  was  by  far  the  least  experienced  of  the  three  subjects  as  far 
as  total  flying  hours  was  concerned,  although  he  has  the  most  familiarity 
with  the  DOBS  equipment  per  se.  The  other  note  regarding  Table  3.8 
is  the  fact  that  there  is  very  little  difference  in  the  two  sigma  FTE 
values  between  the  three  pilots.  On  an  overall  basis  their  performance 
was  quite  consistent.  This  can  also  be  seen  from  an  examination  of 
Table  3.7. 

Table  3.8  DDBS  -  CDI  Aggregation 


Total  All  Flights  Per  Pilot 
(combined  terminal  and  approach  data) 


Pilot 

N 

X 

2a 

A 

3308 

.054 

.986 

B 

732 

-.073 

.636 

C 

893 

-.006 

.876 

Total  All 
Flights 

4933 

.024 

.927 

For  purposes  of  the  objective  comparison  of  FTE  between  the  DDBS 
tests  and  the  baseline  tests  of  Reference  3,  the  test  data  has  been 
aggregated  into  the  terminal  and  approach  phases  of  flight.  Table 
3.9  presents  the  directly  relatable  results  of  these  tests.  As  can 
bo  seen,  the  sample  sizes  between  the  two  tests  are  quite  comparable 
for  each  phase.  While  the  mean  values  are  somewhat  different,  which 
does  have  an  impact  when  total  airspace  utilization  is  considered, 
the  principal  measure  of  comparison,  the  two  sigma  values  of  FTE, 
are  markedly  improved  through  the  use  of  DDBS  in  both  phases  of  flight. 


3-10 


Table  3.9  FTE  Summary 
(Baseline  vs  DDBS) 


Flight  Test 

Phase 

N 

FTE 

[nm) 

X 

2  o 

Baseline 
(Ref.  3) 

Termi nal 
Approach 

5602* 

519** 

-.247 

-.340 

1.540 

1.661 

DDBS 

Terminal 

Approach 

4439J 

494ft 

.027 

-.005 

.950 

.684 

♦SID  and  STAR  to  Hotel  or  Baltic  Waypoint 
♦♦Entire  Final  Approach  Segment  Hotel-India-Map 
and  Baltic-Carolina-Map 

*SID  and  STAR  to  Hotel,  Baltic  or  Bay  Waypoint 
t+Entire  Final  Approach  Segment  Hotel-India-Map, 
Baltic-Carolina-Map  and  Bay-Map 

(See  Figures  2.3  through  2.11) 

In  the  strict  statistical  sense,  the  mere  observation  that  two 
values  are  "markedly  different"  has  no  quantifiable  meaning.  For 
this  reason,  the  variance  between  the  standard  deviations  of  the 
baseline  test  and  the  DDBS  test  was  determined  using  the  variance 
ratio  test.  The  variance  ratio  test  is  defined  as: 

greater  estimate  of  the  variance  of  the  population 

F  - - 

lesser  estimate  of  the  variance  of  the  population 

The  Null  Hypothesis  was  used,  applying  Bessel's  correction  to 
the  variance  to  arrival  at  the  best  estimates.  F  distribution  tables 
were  used1,  J  to  determine  the  probability  level  of  significance  of 
variation.  All  results  of  these  calculations  showed  probability 
levels  of  high  significance,  meaning  that  there  was  only  a  very  small 
chance  that  the  results  could  have  arisen  by  chance.  Table  3.10 
summarizes  these  findings.  It  defines  the  flight  test  data  sets  by 
flight  phase  (approach  or  terminal).  The  probability  level  describes 
the  level  of  significance  or  chance  that  the  result  could  have  arisen 
by  chance,  i.e.,  p  =  .05,  1  chance  in  20;  p  =  .01,  P  =  -005, 

-230-;  p  =  .001,  .  Table  3.11  shows  the  set  of  variables  which 

were  used  for  the  variance  ratio  test  calculations. 
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Table  3.10  FTE  Significance  Values 


Flight  Test 

Phase 

FTE 

FTE 

Sample  Points 

Probability 

Level 

Baseline 

Termi nal 

5602 

DDBS 

4439 

mmm 

Baseline 

Approach 

519 

MB 

DDBS 

494 

■MR 

2  1  2 

The  equation  a,  =  ( - )a,  was  applied  to  the  baseline  and  DDBS 

i  r^-l  1 

flight  test  data  for  each  phase  in  Table  3.11.  Taking  the  terminal 
phase  as  an  example,  the  baseline  is  defined  as  nj  =  5602,  a i  =  .770; 
and  DDBS  is  defined  as  n2  =  4439,  02  =  .475.  Therefore: 

of  =  (  —jy  ).7702  =  .593  with  5601  df. 


In  a  like  manner: 


of  =  (-Zl£l-).4752  *  .226  with  4438  df. 


Then,  by  definition,  the  variance  ratio  F  is: 


F  = 


.593 

.226 


2.62 


Table  3.11  FTE  Variance  Test  Variables 


Flight  Test 

Phase 

N 

0 

Degrees  of  Freedom 
df  (n-1) 

Baseline 

Terminal 

5602 

.770 

5601 

DDB 

4439 

.475 

4438 

Baseline 

Approach 

519 

- 

.831 

518 

DDB 

494 

.342 

493 
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Ttte  number  of  degrees  of  freedom  is  5601  for  the  greater  variance 
estimate  and  4438  for  the  lesser  variance  estimate.  Entering  the  F 

r  91 

distribution  tables1-  J  for  the  variance  ratio  with  these  degrees  of 
freedom  (in  this  case  considered  to  be  infinite  because  of  the  very 
large  values),  the  value  of  F  =  2.62  is  seen  to  be  greater  than  the 
F  values  for  either  the  .005  level  and  the  .001  level.  Therefore, 
the  significance  of  variance  between  the  standard  deviations  is  much 
less  than  the  .001  level  and  is  regarded  as  having  a  probability  level 
of  p  =  .001,  one  chance  in  1  ,000  of  the  observed  result  arising  by 
chance.  The  significance  test  was  similarly  applied  to  the  approach 
phase  data  set,  also  yielding  a  probability  level  of  p  =  .001,  as 
summarized  in  Table  3.10. 

3.4  SUMMARY 

The  results  of  the  DDBS  tests  indicated  a  substantial  improvement 
in  pilot  blunder  performance  as  influnced  by  DDBS.  An  analysis  of 
the  blunder  performance  of  three  identical  subject  pilots  showed  a 
reduction  in  blunders  per  flight  in  an  identical  RNAV  route  structure/ 
test  aircraft  environment  from  0.87  blunders  per  flight  under  standard 
RNAV  conditions  to  0.15  blunders  per  flight  under  DDBS  conditions. 

On  an  overall  pilot  population  basis  the  baseline/DDBS  comparison  was 
1.03  to  0.15  blunders  per  flight,  respectively. 

Pilot  steering  performance  (FTE)  was  similarly  markedly  improved 
through  the  introduction  of  DDBS.  Baseline  test  results  indicated  mean 
and  two  sigma  FTE  results  of  -.247  ±  1.540  nm  for  the  terminal  phase 
as  compared  to  0.27  ±  0.950  nm  for  the  DDBS  tests.  Similarly,  during 
the  final  approach  phase  the  baseline  results  were  -.340  ±  1.661  nm 
as  compared  to  -.005  ±  .684  nm  for  DDBS.  Both  of  these  comparisons 
were  found  to  be  significant  at  the  p  =  .001  probability  level, 
indicating  an  extremely  reliable  quantitative  reduction  in  steering 
error  due  to  the  decreased  cockpit  workload  afforded  by  the  DDBS  concept. 
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4.0 


CONCLUSIONS  AND  RECOMMENDATIONS 


4.1  GENERAL 

■  Ranging  from  the  general  to  the  specific,  this  report  has  deliberately 
attempted  to  consider  only  those  issues  which  bear  directly  on  a 
specific  problem  solution  which  is,  of  course,  only  a  small  contributor 
to  the  overall  solution  of  the  ATC/air  traffic  growth  problem.  Area 
navigation  is  a  concept  of  air  navigation  and  air  traffic  control 
which  has  given  indications  through  study  and  simulation  of  being  able 
to  aid  in  the  alleviation  of  the  problems  of  traffic  congestion,  traffic 
delays,  and  increased  ATC  system  personnel  staffing  requirements.  One 
major  aspect  of  the  implementation  of  the  area  navigation  concept  is 
the  potential  problem  of  increased  cockpit  workload  on  pilot  performance 
as  regards  blunders  and  steering.  The  possibility  of  automating  the 
area  navigation  cockpit  data  entry  function  as  a  means  of  reducing 
cockpit  workload  and  thus  improving  pilot  performance  has  been  advanced 
as  a  potential  solution  to  this  problem. 

4.2  CONCLUSIONS 

The  studies,  simulations  and  experiments  previously  reported  have 
established  several  hypotheses  or  trends  which  resulted  in  three  major 
related  areas  of  pilot/ATC  system  performance  which  needed  substantiation 
either  subjectively  or  quantitatively.  The  results  of  this  DDBS  research 
then,  should  be  considered  in  light  of  the  following  questions: 

1.  Does  the  addition  of  broadcast  data  into  the  RNAV 
environment  substantiate  the  predicted  reduction 
in  cockpit  workload  and  pilot  blunder  performance? 

2.  Does  the  addition  of  digital  broadcast  data  significantly 
improve  the  previously  unsatisfactory  level  of  pilot 
steering  performance  (FTE)  for  a  single  waypoint  general 
aviation  RNAV  system? 

3.  Is  the  DDBS  concept  operationally  acceptable  in  an 
RNAV  terminal  area,  including  the  compatibility  of 
broadcast  navigation  information  with  controller- 
initiated  maneuvers? 
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As  regards  the  pilot  blunders  issue,  a  considerable  amount  of 
research  has  led  to  the  primary  conclusion  that  an^  approach  which 
reduces  the  overall  cockpit  workload  should  have  a  positive  impact  on 
reducing  blunder  rates,  considered  either  on  a  blunder  per  flight  basis 
or  on  a  blunder  per  operation  basis.  Again,  considering  the  constraining 
factor  of  RNAV  systems  and  RNAV  routes/procedures,  several  references 
have  verified  a  reduction  in  blunder  rate  as  a  function  of  increasing 
waypoint  storage  capacity.  Reference  10  observed  a  direct  inverse 
correlation  of  blunder  rate  with  systems  of  waypoint  storage  of  1 ,  2 
and  8  waypoints.  Reference  11  again  documented  a  consistent  decrease 
in  blunders  per  flight  with  waypoint  storage  capacities  of  1,  2,  4  and  8. 

In  addition.  Reference  11  showed,  via  a  side  task  loading  experiment, 
that  the  pilot's  information  processing  rate,  or  his  ability  to  concentrate 
his  attention  on  supplemental  tasks,  was  worse  with  a  two  waypoint 
system  and  best  with  an  eight  waypoint  system.  References  12  and  13 
both  concluded  that  blunder  rate  (blunders  per  operation)  as  well  as 
blunders  per  flight  could  be  substantially  reduced  if  the  cockpit 
workload  situation  were  eased,  i.e.,  by  reducing  the  number  of  operations 
per  flight  required. 

All  of  these  experiments  indicated  that  some  level  of  pilot  blunders 

could  be  reduced  if  the  workload  on  the  pilot  was  reduced.  Conversely, 
ri2i 

it  was  also  shownL  J  that  the  RNAV  environment  as  it  was  originally 
conceived  requires  a  level  of  increased  cockpit  workload  that  could 
adversely  affect  pilot  blunder  performance.  It  was  for  this  reason 
that  the  digital  data  broadcast  concept  was  evolved^’ ^  in  an  attempt 
to  automate  the  pilot's  data  entry  function,  reduce  his  data  input 
errors  and  reduce  his  overall  workload,  thus  potentially  reducing  his 
overall  blunder  rate. 

The  results  of  the  experiments  reported  in  this  document  and  in 
Reference  3  verify  that  the  DDBS  concept  resulted  in  a  dramatic  reduction 
in  pilot  blunders.  In  a  direct  comparison  of  three  identical  subject 
pilots  flying  the  same  routes  in  the  same  aircraft  in  the  same  operational 
flight  environment,  the  use  of  DDBS  resulted  in  a  reduction  of  pilot 
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blunders  from  a  value  of  0.87  blunders  per  flight  for  a  conventional 
single  waypoint  RNAV  system  to  a  value  of  0.15  blunders  per  flight  for 
a  DOBS  RNAV  system.  When  considering  the  total  data  set  including  all 
of  the  subject  pilot  population  the  comparison  was  even  more  striking, 
i.e.,  1.03  blunders  per  flight  for  conventional  RNAV  as  opposed  to 
0.15  for  DOBS.  Since  these  blunder  values  were  obtained  in  a  comparison 
between  two  flight  experiments  that  were  deliberately  designed  to  have 
a  maximum  of  commonality  between  controlled  variables  such  that  the 
conventional  RNAV  vs  DDBS  comparison  was  the  only  difference,  it  is 
considered  that  the  observed  blunder  rate  reduction  can  be  directly 
ascribed  to  the  influence  of  the  DDBS  concept  and  system  implementation. 

The  second  issue,  that  of  pilot  steering  performance,  is  of  similar 

importance  to  the  overall  acceptance  and  implementation  of  area 

navigation  into  the  National  Airspace  System.  Reference  3  has  stated 

that  the  levels  of  flight  technical  error  (FTE)  resulting  from  the 

baseline  testing  of  a  single  waypoint  area  navigation  system  were 

unacceptable  for  certification  purposes.  The  data  contained  in  Reference 

3  verifies  the  fact  that  the  terminal  and  approach  phase  values  of  FTE 

obtained  from  the  "baseline"  RNAV  flights  tests  are  far  above  the  values 

n  5i 

set  as  the  standard  for  FAA  certification1'  .  As  in  the  case  of  pilot 
blunders,  the  automation  of  the  data  input  function  through  the  use  of 
DDBS,  and  its  resultant  decrease  in  pilot  workload,  was  expected  to 
improve  the  pilot  steering  performance  of  the  low  cost  RNAV  user  to 
acceptable  levels^. 

Again  referring  to  the  direct  comparison  between  the  results  of  the 
baseline  experiment  of  Reference  3  and  the  current  DDBS  flight  experiment. 
Table  3.9  has  directly  documented  and  verified  the  initial  assumption 
that  DOBS  can  substantially  reduce  FTE  in  an  RNAV  flight  environment. 

In  the  case  of  the  basic  terminal  phase  of  flight,  the  conventional 
RNAV  FTE  two  sigma  value  of  ±1.540  nm  was  reduced  to  ±0.950  nm  in  the 
case  of  the  DDBS  system.  This  latter  value  is  within  the  nominal  value 
of  1.0  nm  set  for  terminal  phase  FTE  by  AC  90-45A^15^  and  the  RNAV  Task 
Force^.  Similarly  in  the  final  approach  phase  the  conventional  RNAV 
value  of  ±1.661  nm  was  reduced  to  ±0.684  for  DDBS.  While  the  DDBS  value 
for  approach  is  still  above  the  nominal  value  of  0.5  nm^’^,  it  is 
considerably  improved. 


In  considering  the  verification  of  the  reduction  in  FTE  due  to 
DDBS,  the  statistical  significance  of  the  observed  quantitative  values 
must  be  considered.  As  has  been  shown  in  Chapter  3,  the  FTE  improvements 
in  both  terminal  and  approach  phases  of  flight  have  been  shown  to  be 
significant  at  the  p  =  .001  probability  level,  thus  assuring  that  these 
experimental  results  and  differences  can  be  considered  to  be  valid. 

As  regards  the  operational  acceptability  of  the  DDBS  concept  as  an 
element  of  any  future  ATC  system,  this  issue  is  extremely  subjective, 
and  must  be  approached  in  the  context  of  providing  initial  guidelines 
and  observations  at  this  time  rather  than  reaching  absolute  conclusions. 
However,  the  experience  gained  in  the  DDBS  operational  flight  test 
evaluation  did  allow  sfeveral  pertinent  conclusions  to  be  drawn  concerning 
several  pertinent  issues.  A  primary  benefit  of  the  DDBS  concept  resulted 
from  the  decrease  in  pilot  workload  involved  in  RNAV  waypoint  definition. 
This  critical  navigation  input,  particularly  in  the  final  phases  of  the 
terminal  area  transition,  is  very  time-critical  in  conventional  general 
aviation  RNAV  systems: 

4.3  RECOMMENDATIONS 

Thus  far,  several  critical  issues  regarding  the  DDBS  concept  of 
cockpit  input  data  automation  have  been  resolved.  The  technical 
feasibility  of  the  basic  concept  and  the  tested  Engineering  Model  hardware 
has  been  proven.  The  predicted  improvement  in  pilot  blunder  and  steering 
performance  has  been  verified.  The  operational  suitability  of  the  DDBS 
concept  as  an  element  of  the  ATC  system  has  been  demonstrated.  The 
applicability  of  the  results  of  this  research  has  been  identified. 

However,  there  are  still  two  major  areas  of  uncertainty  that  remain 
to  be  clarified  before  a  final  decision  can  be  made  regarding  the  ultimate 
Incorporation  of  the  DDBS  concept  into  the  ATC  system  of  the  future. 

First,  the  DDBS  flight  test  program  was  conducted  with  a  single  DDBS 
equipped  aircraft  flying  in  a  semi -controlled  flight  environment.  It 
still  needs  to  be  demonstrated  that  a  dense  terminal  area  traffic 
environment  can  be  operated  efficiently  with  some  mix  of  DDBS  and  non- 
DDBS  aircraft.  It  is  Important  to  define  whether  or  not  all  aircraft 
entering  a  DDBS  terminal  area  must  be  DDBS  equipped  as  a  "price  of 


admission"  or  whether  this  equippage  is  a  user  option.  As  in  the 
simulations  of  References  7  and  8,  the  logical  source  of  this  data 
would  be  a  real  time  DDBS  terminal  area  simulation  run  on  the  Digital 
Simulation  Facility  at  the  FAA  NAFEC  facility.  As  in  the  previous 
simulations,  the  percentage  of  DDBS  equipped  aircraft  could  be  varied 
from  0%  to  100%,  and  such  parameters  as  controller  talk  time,  controller 
messages,  and  aircraft  delay  time  could  be  recorded.  In  this  manner, 
data  pertinent  to  controller  staffing  studies  could  be  accumulated 
along  with  a  further  evaluation  and  refinement  of  air  traffic  control 
procedures  in  a  DDBS/RNAV  environment  could  be  acquired. 

Second,  and  directly  correlated  to  the  first  issue,  the  degree  of 
implementation  of  the  DDBS  concept,  and  indeed  the  decision  to  embark 
on  the  program  at  all,  is  directly  dependent  on  the  economic  viability 
of  the  concept.  From  the  FAA  point  of  view,  capital  expenditures  will 
be  required  in  order  to  convert  the  current  DME  ground  stations  to  be 
able  to  transmit  DDBS  messages.  The  decision  as  to  which,  or  how  many, 
of  the  950  existing  ground  VOR  stations  should  be  modified  has  not 
been  made.  These  costs  must  be  balanced  off  against  the  potential 
controller  staff  savings  afforded  by  DDBS.  Similarly,  but  more  difficult 
to  quantify  as  far  as  benefits  are  concerned,  the  costs  and  benefits 
to  the  airspace  user  must  be  quantified.  Only  after  this  interrelated 
benefit/cost  ratio  analysis  has  been  performed  can  two  major  questions 
be  answered: 

1.  What  percentage  of  airspace  users  can  be  expected  to  equip 
with  DDBS/RNAV? 

2.  What  are  the  benefit/cost  ratios  for  both  the  FAA  and  the 
airspace  users  for  DDBS  implementation? 

Both  of  these  efforts  must  be  performed  in  conjunction  with  one 
another  before  a  final  evaluation  and  recommendation  regarding  the 
overall  DDBS  implementation  can  be  made. 
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